feat(celery): Add task and worker lifecycle metrics by diogosilva30 · Pull Request #4439 · open-telemetry/opentelemetry-python-contrib

diogosilva30 · 2026-04-15T15:58:57Z

Description

Implements a subset of the Prometheus metrics exposed by Celery Flower directly in the Celery instrumentation, so users who only need basic task/worker observability no longer need to run Flower as a separate service.

Fixes #3458

Type of change

New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)

Changes

Task-level metrics (in `CeleryInstrumentor`)

Metric tracking wired into the existing prerun/postrun/failure/retry handlers:

Flower metric	Impl. name	Labels	Type
`flower_events_total`	`flower.events.total`	task, type, worker	Counter
`flower_task_runtime_seconds`	`flower.task.runtime.seconds`	task, worker	Histogram
`flower_worker_number_of_currently_executing_tasks`	`flower.worker.number.of.currently.executing.tasks`	worker	UpDownCounter (gauge)

Worker lifecycle metrics (new `CeleryWorkerInstrumentor`)

A separate instrumentor that hooks into worker_ready / worker_shutdown / task_received / task_revoked signals in the main worker process:

Flower metric	Impl. name	Labels	Type
`flower_events_total`	`flower.events.total`	task, type, worker	Counter
`flower_worker_online`	`flower.worker.online`	worker	UpDownCounter (gauge)

Registered as a new celery_worker entry point in pyproject.toml.

Omitted metrics

flower_task_prefetch_time_seconds and flower_worker_prefetched_tasks are not implemented. In pool=prefork mode (the default), task_received fires in the main process while task_prerun fires in a child process — there is no shared state to compute the time delta between them. Flower can do this because it consumes Celery events externally in a single process; replicating that inside the worker is not feasible without producer-side instrumentation or protocol changes.

Bug fix — memory leak in `task_id_to_start_time`

Previously, task_id_to_start_time was a class-level dict that was never cleaned up after task completion, growing unbounded over time. It is now instance-scoped and entries are removed in _trace_postrun.

Housekeeping

Type annotations added across __init__.py and utils.py
Typed CeleryGetter properly (Getter[Request])
Null guards added to detach_context / retrieve_context / retrieve_task_id_from_message in utils.py
Replaced bare dict metrics store with typed _CeleryTaskMetrics / _CeleryWorkerMetrics dataclasses

How Has This Been Tested?

Comprehensive unit tests covering all new signal handlers and metric emissions (test_metrics.py, ~1100 new lines)
Integration tests that start a real Celery worker in-process to verify worker_ready / worker_shutdown signals
Edge-case tests: duplicate signals, unknown workers, uninstrument/reinstrument cycles
Pre-commit hooks (ruff lint + format) passing
End-to-end testing with a real Django + Celery project exporting to Grafana via OTLP

Trying it out

A self-contained example project (Django + Celery + Docker Compose + Grafana LGTM) is available at celery-metrics-example (separate branch on the fork, not part of this PR).

Quick start:

git clone -b celery-metrics-example https://github.com/diogosilva30/opentelemetry-python-contrib.git /tmp/celery-example
cd /tmp/celery-example/instrumentation/opentelemetry-instrumentation-celery/example
docker compose up -d

# Exercise every metric path
docker compose run --rm -w /repo django \
    uv run --no-sync \
        --directory instrumentation/opentelemetry-instrumentation-celery/example \
        python exercise_metrics.py

Then open Grafana at http://localhost:3000 (admin/admin) and explore metrics prefixed with flower.*.

Does This PR Require a Core Repo Change?

No.

Checklist:

Followed the style guidelines of this project
Changelogs have been updated
Unit tests have been added
Documentation has been updated

diogosilva30 · 2026-04-21T10:37:28Z

Hey @xrmx @aabmass @lzchen — would appreciate a review when you get a chance!

The CI failures seem unrelated to my changes

Introduce Flower-compatible metrics for Celery task and worker events: - flower.events.total: counter for task-sent, task-received, task-started, task-succeeded, task-failed, task-retried, task-revoked - flower.task.runtime.seconds: histogram of task execution time - flower.worker.number.of.currently.executing.tasks: gauge of in-flight tasks per worker - flower.worker.online: gauge tracking worker online/offline status CeleryInstrumentor (child process) handles task tracing and task-level metrics. CeleryWorkerInstrumentor (main process) handles worker lifecycle signals (worker_ready, worker_shutdown) and main-process events (task_received, task_revoked). Prefetch-time metrics (task_prefetch_time_seconds, worker_prefetched_tasks) are intentionally omitted — Celery splits task_received and task_prerun across processes in prefork mode, making it impossible to compute the delta without external event consumption.

MikeGoldsmith

Thanks for the detailed PR @diogosilva30. A few things before we dive into a full review:

Metric naming — using the flower.* namespace copies Flower's Prometheus metric names, but OTel instrumentations should follow OTel semantic conventions rather than replicate another tool's naming. There aren't Celery-specific semconv metrics yet, so the first step would be to open an issue in opentelemetry/semantic-conventions to propose the relevant metric names. We can then implement against those once they're agreed.

Please split this up — at 1600+ lines this is very hard to review with confidence. I'd suggest at minimum:

Memory leak fix for task_id_to_start_time (small, can merge quickly)
Housekeeping (type annotations, null guards, typed dataclasses)
Task-level metrics
CeleryWorkerInstrumentor (new entry point deserves its own discussion)

New CeleryWorkerInstrumentor — adding a new entry point and instrumentor is a significant API addition. Worth discussing the design in the issue first before implementing.

diogosilva30 · 2026-05-05T15:41:17Z

Hey @MikeGoldsmith , thanks for detailed feedback. Agree with split-up

I've created these two initial PRs:

Memory leak: fix(celery): clear task timing state after postrun #4504
House keeping: fix(celery): type getter and harden utility helpers #4505

Thanks also for the point on metric naming. I checked the existing messaging semantic convention metrics, and I think the safe path is to map only the CeleryInstrumentor metrics that already have a clear semantic match.

Potential mapping for CeleryInstrumentor:

flower.task.runtime.seconds -> messaging.process.duration
task-sent event count -> messaging.client.sent.messages
task-received event count -> messaging.client.consumed.messages

For the remaining metrics, I do not think there is a good existing semantic convention fit today:

flower.worker.number.of.currently.executing.tasks
flower.events.total for task lifecycle states like task-started, task-succeeded, task-failed, task-retried, and task-revoked

So the next course of action should be proposing a specific set of semantic conventions opentelemetry/semantic-conventions like it exists for RabbitMQ inside messaging (here)?

emdneto · 2026-05-12T23:15:34Z

Thanks for the PR!

Just a heads-up: we no longer update CHANGELOG.md directly. The changelog is now generated from changelog fragments using Towncrier.

Please add the appropriate changelog fragment for this change instead of editing CHANGELOG.md manually. You can find the instructions and expected format in CONTRIBUTING.md.

emdneto

@diogosilva30 any pending action here?

github-actions · 2026-06-06T04:34:48Z

This PR has been automatically marked as stale because it has not had any activity for 14 days. It will be closed if no further activity occurs within 14 days of this comment.
If you're still working on this, please add a comment or push new commits.

github-actions · 2026-06-20T04:40:48Z

This PR has been closed due to inactivity. Please reopen if you would like to continue working on it.

diogosilva30 marked this pull request as ready for review April 15, 2026 23:00

diogosilva30 requested a review from a team as a code owner April 15, 2026 23:00

diogosilva30 marked this pull request as draft April 16, 2026 00:10

diogosilva30 force-pushed the feat/celery-task-and-worker-metrics branch from 45f5df3 to 94a36e1 Compare April 16, 2026 15:54

diogosilva30 marked this pull request as ready for review April 16, 2026 15:59

diogosilva30 force-pushed the feat/celery-task-and-worker-metrics branch 4 times, most recently from c6ae222 to a136151 Compare April 21, 2026 10:32

diogosilva30 force-pushed the feat/celery-task-and-worker-metrics branch from 10d220d to a83f1b4 Compare April 23, 2026 10:03

diogosilva30 force-pushed the feat/celery-task-and-worker-metrics branch from a83f1b4 to 2b0c2d5 Compare April 24, 2026 10:53

Merge branch 'main' into feat/celery-task-and-worker-metrics

5b77c2d

MikeGoldsmith requested changes Apr 27, 2026

View reviewed changes

This was referenced Apr 29, 2026

fix(celery): clear task timing state after postrun #4504

Merged

fix(celery): type getter and harden utility helpers #4505

Merged

emdneto reviewed May 22, 2026

View reviewed changes

emdneto added this to Python PR digest May 22, 2026

emdneto moved this to Reviewed PRs that need fixes in Python PR digest May 22, 2026

github-actions Bot added the Stale label Jun 6, 2026

github-actions Bot closed this Jun 20, 2026

github-project-automation Bot moved this from Reviewed PRs that need fixes to Done in Python PR digest Jun 20, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(celery): Add task and worker lifecycle metrics#4439

feat(celery): Add task and worker lifecycle metrics#4439
diogosilva30 wants to merge 2 commits into
open-telemetry:mainfrom
diogosilva30:feat/celery-task-and-worker-metrics

diogosilva30 commented Apr 15, 2026 •

edited

Loading

Uh oh!

diogosilva30 commented Apr 21, 2026

Uh oh!

MikeGoldsmith left a comment

Uh oh!

diogosilva30 commented May 5, 2026

Uh oh!

emdneto commented May 12, 2026

Uh oh!

emdneto left a comment

Uh oh!

github-actions Bot commented Jun 6, 2026

Uh oh!

github-actions Bot commented Jun 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

diogosilva30 commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Changes

Task-level metrics (in CeleryInstrumentor)

Worker lifecycle metrics (new CeleryWorkerInstrumentor)

Omitted metrics

Bug fix — memory leak in task_id_to_start_time

Housekeeping

How Has This Been Tested?

Trying it out

Does This PR Require a Core Repo Change?

Checklist:

Uh oh!

diogosilva30 commented Apr 21, 2026

Uh oh!

MikeGoldsmith left a comment

Choose a reason for hiding this comment

Uh oh!

diogosilva30 commented May 5, 2026

Uh oh!

emdneto commented May 12, 2026

Uh oh!

emdneto left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jun 6, 2026

Uh oh!

github-actions Bot commented Jun 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

diogosilva30 commented Apr 15, 2026 •

edited

Loading

Task-level metrics (in `CeleryInstrumentor`)

Worker lifecycle metrics (new `CeleryWorkerInstrumentor`)

Bug fix — memory leak in `task_id_to_start_time`